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necessarily  the  best  available  for  the  purpose. 
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Abstract 

This  document  describes  a research  project  to  recommend  a decision  tool  that  the  Manufacturing  Systems 
Integration  Division  (MSID)  could  use  during  its  strategic-planning  process  to  evaluate  which  standards 
activities  to  support.  This  paper  describes  the  criteria  selected  to  give  priority,  in  National  Institute  of 
Standards  and  Technology  (NIST)  and  MSID  terms,  to  the  standards  activities  that  NIST  and  MSID  do,  or 
should,  support.  There  are  criteria  for  judging  the  quality  of  the  standard  itself,  independent  of  how 
applicable  to  the  NIST  mission.  Other  criteria  are  designed  to  help  MSID  decide  whether  or  not  to 
participate  in  a standard's  development.  A computer-aided,  decision-analysis  tool  was  used  to  show  that 
computer  assistance  would  be  useful  to  make  decisions  when  many  variables  impact  that  decision. 

Key  Words: 

Information-technology  standards,  criteria  evaluation,  NIST  mission  criteria 

1 Introduction 

The  standards  road-map  project  is  part  of  the  Manufacturing  Enterprise  Integration  Project  of  the 
Manufacturing  Systems  Integration  Division  [THOM97],  MSID  is  a division  of  the  Manufacturing 
Engineering  Laboratory  (MEL),  one  of  the  seven  National  Institute  of  Standards  and  Technology  (NIST) 
laboratories.  The  MSID  mission  is  to  promote  economic  growth  by  working  with  industry  to  develop  and 
apply  technology,  measurements,  and  standards  for  information-based  manufacturing.  This  is  accomplished 
by  working  with  the  U.S.  manufacturing  industry,  software  suppliers,  systems  integrators,  and  standards- 
development  organizations. 

The  purpose  of  this  research  project  is  to  recommend  a domain,  relevant  to  MSID,  of  enterprise-integration- 
related  standards,  and  to  indicate  what  is  to  be  included  and  what  is  to  be  excluded  from  that  domain. 
Enterprise-integration-related  standards  are  those,  at  any  level  in  the  enterprise,  that  help  processes  to 
interoperate.  Assume,  however,  that  all  standards  considered  will  be  in  the  information-technology 
category.  There  are  three  document  deliverables  that  are  required  to  satisfy  the  objective.  In  addition  there 
are  written  summaries  of  findings. 

The  three  project  documents  described  below  comprise  the  deliverables  of  this  project. 

1 . The  Criteria  Analysis  describes  criteria  to  help  MSID  justify  participation  in  a particular  standard. 

[This  document] 

2.  A Standards  Baseline  [unpublished]  describes  current  MSID  standards  activities.  This  is  a logical 
predecessor  to  a standards-information  resource,  called  the  standards  landscape,  that  would  include  a 
list  of  sources  and/or  the  standards  by  category  in  the  enterprise-integration-related-standards  domain. 
The  landscape  would  also  include  the  purpose  of  the  standard,  and  the  status  of  standard  development. 
This  resource  could  be  a database,  a World-Wide-Web  site,  a paper  document,  or  a combination  to  be 
determined.  The  information  resource  is  to  be  completed  in  later  work. 
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3.  The  Standards  Classification  Strateg}’  and  Methodology > summarizes  findings,  recommends  a 

methodology  to  identify'  needed  standards,  and  recommends  a schema  to  classify  relevant  standards. 
[NELL99]  ^ 

This  document  is  the  Standards  Road-Map  Project— Criteria  Analysis.  The  purpose  of  this  document  is  to 
describe  an  approach  to  help  MSID  ascertain  what  standards  apply  to  a certain  manufacturing  domain,  to 
offer  decision  support  for  determining  to  what  degree  should  MSID  support  the  development  of  those 
standards,  and  the  justification  for  the  support.  The  Standards-lnformation  Resource,  the  data  tool  that 
would  provide  information  to  support  decision-making  is  a work  in  progress.  The  information  resource,  a 
database  such  as  Access,  or  a World-Wide-Web  site  using  hyperlinks  to  the  standards  information,  is  a 
compendium  of  information  about  standards  relevant  to  the  enterprise-integration  domain.  The  Standards 
Classification  Strategy  and  Methodology  document  has  been  completed  [NELL99],  That  document  defines 
categories  for  the  standard  set  and  ways  to  both  expose  and  prune  the  search  space.  The  total  search  space  is 
considerably  larger  than  the  amount  of  resources  available  to  support  such  a search  task,  even  when  using 
computer-assisted  tools. 

Standards  development  is  a labor-intensive  process.  In  the  information-technology  domain  many  of  the 
standards  that  are  perceived  to  be  needed  to  improve  enterprise  integration  are  leading  technology  before 
that  technology  has  been  developed  fully.  Therefore,  the  standards-development  process  is  additionally 
costly  because  elements  of  the  technology  are  being  developed  concomitantly  with  the  standards.  With 
limited  resources,  choosing  which  standards  activities  to  support  and  how  much  support  is  appropriate  is 
more  difficult  without  some  criteria  that  would  rank  the  various  standards  developments  with  respect  to 
their  value  to  the  MSID  and.  ultimately,  its  customers. 

2 Standards-evaluation  process 

A standard  is  a documented  agreement  about  rules,  guidelines,  criteria,  or  definitions  regarding  a product, 
process,  or  service.  Standards  apply  to  a product,  process,  or  service  only  in  a context  and  to  the  extent  that 
the  standard  serves  its  function;  that  is,  it  is  apropos,  of  high  quality,  and  is  easy  to  use.  For  example,  a 
standard  specify  ing  interfaces  requiring  a user  to  configure  operations  in  such  a way  that  is  not  productive 
for  that  user,  will  not  have  utility  and  probably  will  not  be  used. 

2.1  Quality  of  the  standard 

Before  determining  whether  a particular  standard  should  be  supported  and  to  what  degree,  an  analysis  to 
evaluate  if  that  standard  itself  is,  or  will  be,  a good  document  would  be  prudent.  The  Institute  for  Defense 
Analysis  [ROBY96]  has  developed  a model  to  provide  a user  with  specific  information  about  a standard. 
This  information  is  intended  to  facilitate  decisions  about  whether  a standard  can  meet  specific  needs. 

The  IDA  standards  evaluation  model  consists  of  15  criteria  grouped  into  four  categories:  quality,  standards 
support,  stability,  and  marketplace. 

• Quality,  completeness,  technical  quality,  unambiguous  expression,  clarity,  strictly  defined  interface, 
and  flexibility. 

• Standards  support : credentials  of  the  standard's  approving  body,  standard's  scope  of  acceptance,  and 
standard's  conformance  specification. 

• Stability,  stability  of  domain,  life  cycle  of  the  standard,  stability  of  the  standard. 

• Marketplace : number  of  acceptable  products,  marketplace  presence,  cost  of  standard. 


Usage  issues ; 

One  should  be  cautious  about  using  a particular  rating  in  the  IDA  analysis.  A high  or  a low  rating  in  this 
analysis  does  not  indicate  whether  MSID  should  or  should  not  change  its  emphasis  toward  supporting  a 
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particular  standard.  A high  score  could  indicate  that  the  standard  is  important  and  relevant  to  industry, 
government,  and  NIST,  and  deserves  considerable  MSID  support.  Alternatively  a high  score  could  mean 
that  the  standard  is  mature  and  stable  and  that  NIST  development  support  is  unnecessary. 

A low  score  could  indicate  that  the  standard  is  not  needed  and  that  the  standard  should  not  be  replaced.  Or, 
a low  score  could  indicate  that  the  standard  is  not  needed  and  there  is  need  for  a replacement.  Therefore, 
results  of  this  analysis  should  be  accompanied  with  some  explanatory  notes  to  explain  the  reason  for  the 
score  assigned  to  the  standard. 

Information  generated  by  this  analysis  should  be  used  in  an  objective-analysis  environment  only.  A low 
rating  in  these  criteria  makes  a statement  about  the  standard  itself,  and  only  within  the  context  of  the 
assumptions  and  conditions  at  the  time  of  the  analysis.  The  information  must  be  applied  knowing  how  the 
standard  should  be  used  and  by  whom.  Perhaps  some  of  the  criteria  are  not  as  important  as  are  others  in  a 
particular  context.  Also,  the  information  should  not  be  a major  determinant  about  whether  NIST  should 
support  the  standard.  Rather,  NIST  should  use  the  criteria  discussed  above  to  form  notes  that  point  to  areas 
where  NIST  could  concentrate  to  help  create  a better  standard— should  the  analysis  described  below  indicate 
that  NIST  support  is  warranted. 

2.2  Objectives  and  viewpoints  of  the  standards 

The  International  Organization  for  Standardization  (ISO)  has  stated  the  objectives  of  standardization,  in  the 
ISO  directives,  part  2,  section  5 and  also  in  ISO  TR  10314  [1 SO90] [I S09 1 ] : 

• Mutual  understanding 

• Health,  safety,  protection  of  environment 

• Interface,  interchangeability 

• Fitness  for  purpose 

• Variety  control 

ISO  TR  10314  [ISO90][ISO91  ] also  has  selected  several  viewpoints  that  a standard  may  take  in  a 
manufacturing  enterprise: 

• Safety 

• Environment 

• Compatibility 

• Performance 

• Operability 

• Maintainability 

• Reliability 

• Qualifications 

• Description 

In  evaluating  need,  NIST  must  in  turn  evaluate  the  nature  of  the  standard  being  considered  to  be  sure  that 
the  objective  and  viewpoint  of  the  standard  are  in  the  NIST  purview. 

3 Standards  road-map  project-selecting  the  criteria 

One  objective  of  this  project  is  to  use  certain  criteria  selected  by  the  project  to  allow  MSID  to  help 
determine  whether  or  not  MSID  should  assign  resources  to  support  a particular  standards-development 
activity.  The  criteria  are  intended  to  add  priority-setting  capability  to  the  information  resource  and  to 
facilitate  recommendations  to  MSID  about  which  standards  to  support  and  why. 

3.1  Rationale  for  choosing  the  criteria 

In  general,  the  relevance  and  priorities  of  MSID  participation  in  the  manufacturing  domain  would  parallel 
the  relevance  and  priorities  of  NIST  work  for  US  industry  in  general  [BELL094],  Indeed,  the  flavor  of  the 
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NIST  laboratories'  priorities  was  the  basis  for  defining  criteria  for  the  standards  road-map  project,  and  the 
nature  of  these  criteria  can  be  seen  in  the  Technical-Approach  section  of  the  standards  road-map  project 
plan  [THOM97].  Thus,  the  criteria  for  MSID  participation  in  standards-development  activities  were 
developed  by  adapting  the  for  the  NIST  laboratories'  priorities  into  a context  that  is  more  directly  applicable 
to  criteria  that  help  manufacturing  enterprises.  The  project  named  the  adapted  criteria  applicability > criteria. 

The  purpose  of  the  applicability  criteria  is  to  encourage  users  to  know  and  anticipate  NIST  customer  needs 
over  the  next  five  years.  The  criteria  will  be  used  by  MSID  to  evaluate  a standard  and  its  relevance  to  our 
industrial  customer's  manufacturing  work. 

Then,  having  determined  the  applicability  of  a standard  to  MSID  participation,  the  information  provided  by 
the  applicability  criteria  should  be  used  to  help  decide  whether,  and  how  much,  support  is  appropriate;  or 
whether  NIST  support  should  be  curtailed  or  not  started.  The  additional  criteria  that  guide  this  process  have 
been  defined  and  have  been  called  decision  criteria.  These  decision  criteria  were  given  to  the  project  by 
MSID  management  at  the  beginning  of  the  project  and  are  presented  in  the  Needs-Addressed  section  of  the 
project  plan  [THOM97], 

3.2  Organizing  the  criteria 

As  stated  in  3.1,  the  criteria  selection  began  by  considering  the  six  guiding  criteria  that  the  NIST 
laboratories  use  to  set  priorities  in  their  strategic  planning  [BELL094],  These  six  criteria  are  intended  to 
guide  decisions  regarding  whether  NIST  should  engage  in  some  activity.  For  this  project,  these  applicability 
criteria  are  intended  to  reflect  the  relevance  of  MSID  participation  in  the  development  of  a standard.  The 
specific  criteria  are  described  in  section  4 of  this  report.  These  criteria  are: 

• Magnitude  and  immediacy  of  industrial  need. 

• Degree  of  correspondence  between  a particular  industrial  need  and  the  NIST  mission  to  develop 
infrastructure  technologies. 

• Opportunity  for  NIST  participation  to  make  a major  difference. 

• Nature  (the  quality  of)  and  size  of  the  anticipated  impact  resulting  from  NIST  participation. 

• NIST  capability  to  respond  in  timely  fashion  with  high-quality  solutions. 

• Nature  (support  of  process  and  infrastructure  technology)  of  opportunities  afforded  by  recent  advances 
in  science  and  technology. 

MSID  management  asked  that  these  six  criteria  be  grouped  into  levels  that  were  specified  at  the  outset  of 
the  road-map  project  and,  they  were  presented  in  the  Technical-Approach  section  of  the  project  plan 
[THOM97],  Rather  than  interpret  these  items  as  levels,  the  project  saw  them  as  categories  into  which  the  six 
criteria  could  be  arranged.  The  categories  are  listed  below: 

• Category  1 : Setting  priorities  and  measuring  results  at  NIST 

• Category  2:  Relevance  for  MSID  involvement 

• Category  3:  Does  or  could  NIST  use  the  standard-This  category  became  a criterion  as  well. 

The  decision  criteria  specified  at  the  outset  of  the  project  by  division  management  to  try  to  quantify  the 
needed  support  for  a standard  are: 

• What  kind  of  standard  is  it? 

• Should  NIST  support  the  standard  development  with  people? 

• How  many  people  will  it  take  to  be  effective? 

The  applicability-criteria  and  decision-criteria  wording  has  been  slightly  modified  in  4 to  be  more  directly 
applicable  to  MSID  work  with  manufacturing  enterprises  and  with  standards  affecting  enterprise  operation. 

4 Analyzing  and  quantifying  the  criteria 
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Users  of  the  methodology  were  to  identify  a subject  standard  activity  and  assign  a numerical  value  to  that 
standard  activity  depending  on  how  well  the  standard  activity  satisfied  a criterion.  Because  the  differences 
among  various  activities  would  be  largely  subjective  and  qualitative,  a coarse  numerical  scale  was  selected 
rather  than  a fine  one.  The  first  trial  assigned  values  of  small,  medium  and  large  to  the  ratings.  A value  of 
five  for  large,  three  for  medium,  one  for  small  or  nil  was  selected.  While  values  of  one  through  ten  and  one 
through  one  hundred  were  considered,  being  able  to  distinguish  between,  say,  a score  of  sixty  and  a score  of 
fifty  three,  would  be  meaningless  and  have  no  statistical  relevance. 

Having  a rigid  scoring  system  would  imply  that  there  is  a hurdle  score.  Ratings  above  the  hurdle  would 
receive  MSID  support  and  ratings  below  that  would  not.  This  would  be,  as  with  investment  opportunities  in 
organizations  with  rigid  hurdle  rates,  very  unfortunate.  Any  valuation  based  on  qualitative  judgements  that 
receives  a quantitative  score  must  be  judged  more  on  a case-by-case  analysis  and  organizational  mission 
than  the  score.  Managers  seeking  to  relinquish  their  judgement  process  with  rigid  tools  such  as  this  would 
do  better  in  an  occupation  with  a more  numerically  oriented  structure. 

Appendix  A lists  the  criteria  categorized  in  3.2,  above,  with  abbreviated  titles.  Preliminary  values  are  given 
as  an  example.  The  values  have  been  assigned  to  each  criterion  relating  to  the  degree  of  compliance  judged 
for  each  standard.  These  values  have  been  assigned  numbers: 

• The  first  corresponds  to  the  weakest  applicability,  and  assigned  a number  1 

• The  second  corresponds  to  a medium  applicability,  and  assigned  a number  3 

• The  third  corresponds  to  the  strongest  applicability,  and  assigned  a number  5 

The  values  listed  this  way  for  these  criteria  can  lead  to  curious  results,  as  the  criteria  analysis  and 
interpretation  discussed  below  indicates.  In  many  cases  the  criteria  need  an  associated  possible  MSID 
position  to  state  whether  the  low  or  the  high  value  is  better  for  MSID,  either  politically,  organizationally, 
technically,  or  to  satisfy  an  industrial  need.  Sometime  the  lowest  and  highest  are  not  desirable,  thus  the 
middle  might  be  preferred.  The  problem  is  that  resulting  criteria  values  depend  on  the  MSID  position 
attached,  and  that  position  must  be  reviewed  every  time  the  process  is  used.  The  position  depends  on  the 
current  environment  that  could  change  on  very  short  notice.  Please  refer  to  Appendix  B for  possible 
interpretations  of  the  criteria. 

Another  issue  is  that  the  criteria  are  not  orthogonal,  in  the  sense  that  a change  made  to  the  value  of  some  of 
them  has  an  effect  on  the  value  of  some  of  the  others.  Appendix  C has  been  constructed  to  illustrate  the 
interrelationships  among  the  criteria. 

4.1  Criteria  listing 

The  standards-road-map  criteria  listed  above  in  3.2  are  organized  into  the  three  categories,  also  mentioned 
in  3.2.  Appendix  D is  a complete  listing  of  the  criteria.  The  criteria  listed  below  have  been  modified  slightly 
to  make  them  directly  applicable  to  the  standards-development  activities  of  MSID. 


Applicability  criteria 
Category  1 

1 . Magnitude  and  immediacy  of  industrial  need 

2.  Nature  and  size  of  anticipated  impact 

3.  Nature  of  opportunity  afforded 

Category  2 

4.  Degree  of  correspondence  between  industrial  need  and  MSID  mission 

5.  Opportunity  for  MSID  to  make  a difference  (covered  in  criterion  4.2  and  becomes  criterion  2,  see 
Appendix  E 
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6.  Ability  to  anticipate  and  respond  in  timely  fashion 
Category  3 

7.  Does  or  could  NIST  use  the  standard  (No  breakdown  recommended) 

Decision  criteria 

8.  The  nature  of  the  standards  activity  (No  breakdown  recommended) 

9.  The  need  for  NIST  resources  (No  breakdown  recommended) 

10.  The  bang-for-the-buck  (No  breakdown  recommended) 

4.2  Criteria  analysis 

After  the  discussion  of  each  criterion,  the  criterion  is  stated  in  a more  precise  way  by  dividing  it  into 
segments  of  meaning.  For  example,  the  criterion  1 . magnitude  and  immediacy  of  industrial  need , has  two 
main  segments:  the  magnitude  and  the  immediacy.  Each  of  these  segments  can,  in  turn,  be  divided  into  sub 
elements.  The  segments  are  numbered  to  make  it  easier  to  refer  to  them.  Appendix  E shows  a graphic 
organization  of  the  criteria  and  how  they  decompose  into  sub  elements. 

Usage  issues  must  be  considered  when  applying  many  of  these  criteria  to  better  interpret  what  some  of  the 
results  could  mean.  Some  results  are  ambiguous.  The  usage  issues  are  intended  to  lead  a strategic  planner 
toward  a statement  of  a possible  MSID  position  relative  to  the  criterion.  Appendix  B offers  a summary  of 
possible  interpretations  when  evaluating  the  criteria  with  respect  to  standards  activities. 

Finally,  some  assumptions  are  suggested  to  help  guide  the  thinking  process  associated  with  interpreting 
what  a particular  value  for  a criterion  is  assumed  to  mean. 

4.2.1  Applicability  criteria 

Category  1:  Setting  priorities  and  measuring  results 
Criterion  1:  Magnitude  and  immediacy  of  industrial  need: 

Description : This  criterion  arises  from  the  basic  way  that  NIST  and  MSID  determine  and  justify  much  of 
their  work  [BELL094],  Industrial  needs  come  from  continuous  interaction  with  industrial  associates  that 
are  directly  involved  in  the  technology  and  with  those  who  control  the  investment  funds  for  such  projects. 
There  are  two  dimensions  to  this  criterion;  the  size  of  the  stated  need  and  the  amount  of  time  available 
before  solutions  must  be  implemented. 

Criterion  1 breakdown: 

1 Magnitude  and  immediacy  of  industrial  need 

1 . 1 Magnitude  of  industrial  need 

1.1.1  Nature  of  entity  expressing  need 

1 . 1 . 1 . 1 Congress-new  law 

1 . 1 . 1 .2  Industrial— segment 

1.1. 1.3  Industry— general 

1 .2  Immediacy  of  industrial  need 

1 .2. 1 NIST  anticipated  the  need  (therefore  no  stated  need) 

1 .2.2  Ability  to  respond  to  stated  need 

1 .2.2. 1 Industry  needs  benefit  in  less  than  three  years 

1 .2.2.2  Industry  needs  benefit  in  less  than  five  years 

Values  assigned : Magnitude  and  immediacy  of  industrial  need  for  standards  that  improve  process 
interoperability  and  information-technology  infrastructure.  This  criterion  has  two  dimensions:  time  and 
stated  need.  The  time  dimension  will  be  assigned  a 1 = no  time  stated,  3 = need  within  5 years,  or  a 5 = 
need  within  3 years.  The  need  dimension  will  be  assigned  a 1 = no  need  stated,  3 = need  qualified  by 
industry  type  or  size,  5 = general  industry  need. 
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Usage  issues:  The  term  need  must  be  defined  on  a case-by-case  basis.  For  example  is  need  immediacy  best 
expressed  by  critical  industries,  most  competitive  companies,  least  competitive,  most  endangered  industries, 
largest,  small-to-medium  sized,  or  most  politically  acceptable?  Do  other  relevant  need  categories  apply? 

Does  the  need  magnitude  mean  that  the  highest  quality  or  most  effective  NIST  support  impacts  companies, 
industries,  individual  workers,  balance  of  trade,  or  the  national  economy? 

Receiving  an  explicitly  stated  need  for  a particular  standard  from  industry  is  difficult  because,  in  our 
opinion,  industry  has  no  direct  need  for  standards  and  there  will  be  no  consistent  answer  about  standards 
from  specific  data  points  in  industry.  In  addition,  industry  really  does  not  have  an  opinion,  people  within  the 
industry  do,  and  that  opinion  is  most  relevant  in  the  context  of  their  particular  function.  Most  people  in 
industry  are  fairly  consistent  about  stating  industrial  needs  such  as  lower  cycle  time,  improved  productivity, 
better  agile  performance,  and  lower  product  cost.  Standards  are  one  way  to  help  achieve  some  or  all  of 
these. 

For  that  reason,  MSID  must  be  able  to  translate  standards  development  as  one  of  many  solutions  to  the 
kinds  of  needs  normally  expressed  by  industry.  MSID  must  then  balance  resources  so  that  the  need  for 
standards  is  resolved  simultaneously  with  other  needs  that  can  improve  industrial  climate.  There  are  many 
others,  such  as  better  metrology,  improved  processes,  improved  infrastructure,  and  employee  training. 
Continued  interactions  with  persons  in  industry  responsible  for  these  improvements  and  carefully  inferring 
the  need  for  each  of  the  various  standards  that  apply  are  imperative  activities  for  MSID  people  interested  in 
standards. 

Another  issue  is  whether  no  stated  need  is  better  than  an  immediate  need  stated  by  many  companies.  No 
stated  need  could  reflect  MSID  anticipatory  focus  is  sharp  or  that  crunch  time  has  not  yet  arrived.  Or,  no 
stated  need  could  mean  that  the  crisis  is  past  and  NIST  is  too  late  to  be  effective. 

A high  and  immediate  need  also  could  mean  that  MSID  anticipatory  focus  is  ineffective,  MSID  is  too  late, 
and  that  many  industries  have  perceived  the  need  and  are  desperate.  Whether  or  not  MSID  should  enter  the 
process  at  this  time  will  certainly  depend  on  MSID  resource  capability  to  work  under  pressure  in  situations 
such  as  this. 

Possible  MSID  position:  MSID  will  meet  or  anticipate  urgent  industrial  needs,  which  if  resolved  would 
enable  integration  improvements. 

Assume: 

• Industry  will  recognize  and  state  needs  that  exist 

• Needs  will  be  stated  in  a consistent  way 

• Industry  will  be  as  effective  recognizing  incipient  needs  as  NIST  is;  therefore,  no  statement  of  need 
means  there  is  not  and  will  not  be  any  need 

• Addressing  a long-term  industrial  need  satisfies  the  anticipation  objective 

Criterion  2:  Nature  and  size  of  anticipated  impact 

Description:  As  part  of  the  continuous  review  process  mentioned  above,  MSID  should  sort  the  things 
industry  could  best  do  for  themselves,  such  as  process  improvements;  from  the  things  a government  can  do 
to  help,  such  as  infrastructure  improvements.  Other  considerations  in  this  category  have  to  do  with  the  status 
of  the  standard  being  considered;  for  example,  the  standard  may  be  mature  and  needs  replacing,  mature  and 
seemingly  serving  its  purpose,  developing  and  needs  consensus  improvement,  or  still  in  a conceptual  stage 
of  a needed  standard.  A related  consideration  is  whether  an  existing  or  alternate  technology  could  be  used. 

Criterion  2 Breakdown: 

2 Nature  and  size  of  anticipated  impact 

2.1  Nature  of  impact 
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2.1.1  Nature  of  support  needed 

2. 1.1.1  Develop  tools 

2. 1.1. 2 Develop  standard 

2. 1 . 1 .3  Develop  test,  conformance 

2. 1 . 1 .4  Develop  capability  that  uses  standard 

2. 1 . 1 .5  Standards  administration 

2.2  Size  of  impact 

2.2.1  Impact  in  small  part  (<20%)  of  problem  (stated  need) 

2.2.2  Impact  in  large  part  (>20%)  of  stated  need 

Values  assigned'.  Nature  and  size  of  the  anticipated  impact  of  MS1D  activities  on  integration  standards  and 
consensus  for  industrial-information  infrastructure.  This  criterion  will  be  assigned  a 1 = slight  impact,  3 = 
indirect  impact,  or  a 5 = direct  impact.  This  criterion  is  similar  to  criterion  5. 

Usage  issues:  The  term  nature  must  be  defined  or  assumed  on  a case-by-case  basis.  Does  nature  of  impact 
refer  to  the  quality  of  involvement  or  type  of  involvement  such  as  funding,  administrative,  technical,  or 
leadership? 

An  assigned  value  indicating  slight  impact  could  be  due  to  only  one  small,  but  key,  area  needing  MSID 
support,  whereas  a high,  or  direct,  impact  rating  could  create  a risk  of  the  output  becoming  a government  or 
NIST  standard. 

Possible  MSID  position:  MSID  will  identify  and  make  significant  impact  in  infrastructural  standard 
domains  where  needed. 

Assume: 

• Nature  of  impact:  MSID  will  be  able  to  improve  the  situation  by  applying  NIST  resources 

• Size  of  impact:  NIST  will  succeed  if  its  efforts  will  improve  significantly  [?]  the  implementation  date 
of  the  standard 

• NIST  will  impact  infrastructure  improvements  rather  than  process  improvements 

Criterion  3:  Nature  of  opportunities  afforded: 

Description:  This  is  related  to  the  criterion  2,  but  is  more  about  the  nature  of  the  MSID  support  needed. 
This  item  is  more  pointed  toward  helping  to  transfer  things  such  as  the  technology  or  knowledge  among 
industry  users  to  accelerate  or  enable  standard  acceptance. 

Criterion  3 Breakdown: 

3 Nature  of  opportunity  afforded 

3.1  Technology  transfer 

3.1.1  Documents 

3.1.2  Seminars 

3.1.3  Co-participation 

3.2  Knowledge  transfer 

3.2.1  Books 

3.2.2  Knowledge  base 

3.3  Publicity 

3.3.1  Mass 

3.3.2  Focused 


8 


Values  assigned : Nature  of  opportunities  afforded  by  recent  advances  in  science  and  technology;  such  as, 
consensus-building  workshops  and  conferences,  technology-transfer  projects,  and  needed  standards.  This 
criterion  will  be  assigned  a 1 = not  relevant,  3 = average,  or  a 5 = highly  applicable. 

Usage  issues : No  issues 

Possible  MSID position : MSID  will  transfer  technology  and  knowledge  to  industry  about  standards 
research,  development,  implementation,  and  testing. 

Assume : MSID  will  find  methods  to  accomplish  effective  technology  and  knowledge  transfer. 

Category  2:  Relevance  for  MSID  involvement 

Criterion  4:  Degree  of  correspondence  between  industrial  need  and  MSID  mission 

Description : The  criterion  implies  that  an  MSID  mission  is  to  develop  standards  that  support 
manufacturing-enterprise,  information-technology  infrastructure.  Therefore,  the  criterion  focuses  on  the 
correspondence  between  the  perceived  need  and  process  interoperability.  The  necessary  exchanges  among 
processes  can  be  inter  company  or  intra  company  (where  company  = legal  entity).  This  is  intended  to 
improve  the  degree  of  interface  definition,  interchange  format,  or  other  process  interaction,  rather  than  to 
improve  the  aspects  of  a particular  process.  Working  with  particular  process  improvements  is  deemed  to  be 
something  industry  should  support  rather  than  government. 

Criterion  4 Breakdown: 

4 Degree  of  correspondence  between  industrial  need  and  MSID  mission 

4. 1 Industry  mission 

4.2  MSID  mission 

4.2.1  Stated  industry  need  (criterion  1) 

4.2.2  Availability  of  resources  (criterion  6) 

4.2.3  Nature  of  opportunity  afforded  (criterion  3) 

4.2.4  Nature  and  size  of  anticipated  impact  (criterion  2) 

4.3  Other  mission 

Values  assigned : Degree  of  correspondence  between  a manufacturing-process-interoperability  need  and  the 
MSID  mission  to  develop  standards  that  support  manufacturing-enterprise,  information-technology 
infrastructure.  This  criterion  will  be  assigned  a 1 = slight  correspondence,  3 = indirect  correspondence,  or  a 

5 = direct  correspondence. 

Usage  issues:  It  is  important  to  determine  whether  meeting  a US  industry  need  is  more  important  than 
MSID  mission,  should  the  two  conflict.  Almost  by  definition,  however,  US  industrial  needs  will  correspond 
to  MSID  mission  regardless  of  what  the  need  is  and  the  MSID  ability  to  satisfy  that  need.  Otherwise,  the 
sense  of  this  criterion  is  meaningless. 

Possible  MSID  position:  MSID  will  assign  resources  to  meet  US  industrial  need  to  the  degree  it  can;  for 
example,  with  respect  to  priorities  provided  by  a standards  road  map,  and  the  appropriateness  of  the  need  as 
inferred  from  analyzing  these  criteria. 

Assume:  That  the  MSID  mission  will,  in  some  way,  correspond  to  US  industry  needs. 

Criterion  5:  Opportunity  for  MSID  to  make  a difference: 

Description:  Another  interpretation  of  the  MSID  mission  is  to  improve  the  state-of-the-art  of  information 
technology  and  information-technology  standards.  These  are  considered  to  be  infrastructural  in  nature  and 
thus  highly  applicable  for  MSID  participation. 

Criterion  5 Breakdown:  None  recommended. 
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I 'ahies  Assigned:  Opportunity  for  MS1D  participation  to  make  a difference  in  SOTA  for  information- 
technology  infrastructure  and  standards.  This  is  affected  by  the  timeliness  of  NIST  reaction.  This  criterion 
will  be  assigned  a 1 = slight  opportunity,  3 = indirect  opportunity,  or  a 5 = direct  opportunity.  This  criterion 
is  too  similar  to  criterion  2 to  be  separate. 

Usage  issues:  No  issues. 

Possible  MSID position:  MS1D  will  identity  and  make  significant  impact  on  infrastructural  standards  where 
needed.  (Same  as  criterion  2) 

Assume:  Criterion  2 will  subsume  this  criterion. 

Recommendation:  Combine  this  criterion  within  number  2 either  totally,  or  add  any  sub-meaning  element 
from  this  criterion  to  the  second. 

Criterion  6:  Ability  to  anticipate  and  respond  in  timely  fashion: 

Description:  A key  part  of  the  NIST  mission  is  to  anticipate  US  industrial  needs  before  they  become 
critical.  In  the  MSID  domain  this  would  apply  to  infrastructural-information  technology.  The  degree  that  the 
need  perceived  is  anticipatory  or  reactive  is  the  essence  of  this  criterion.  The  criterion  applies  to  aspects  of 
an  ability  to  perform,  the  time  until  a usable  standard  can  be  realized,  and  the  appropriateness  of  the  MSID 
resources  available  to  work  on  the  problem 

Criterion  6 Breakdown: 

6 Ability  to  anticipate  and  respond  in  timely  fashion 

6.1  Human  resources  available 

6.1.1  Degree  of  management  support  or  commitment 

6.1.2  Current  NIST  goals 

6. 1 .3  Current  fiscal  climate 

6. 1 .4  Expertise  level  of  resource 

6.2  Standard  status--Availability  of  standard 

6.2.1  Short  time  until  use 

6. 2. 1.1  Committee  draft  exists 

6.2. 1 .2  Draft  standard  prepared  and  balloted 

6.2.2  Long  time  until  use 

6.2.2. 1  Work  not  yet  started— NIST  to  help  generate  consensus  re  need  for  standard 

Values  assigned:  The  MSID  capability  to  anticipate  and  respond  in  timely  fashion  with  high-quality 
solutions.  The  degree  of  high  quality  can  be  defined  only  by  the  users  who  have,  or  will  have,  the  need.  This 
capability  will  be  affected  by  the  priorities  involved  with  availability  of  NIST  staff.  This  criterion  will  be 
assigned  a 1 = reactive  response,  3 = with  technology,  or  a 5 = anticipating  technology. 

The  other  part  of  the  criterion  is  the  time  it  will  take  to  produce  the  standards  effect  desired.  This  could 
result  in  a 1 = no  way  to  have  a standard  in  time,  3 = just  make  the  time  requirement,  and  5 = Beat  the  time- 
-and  anticipate  the  need,  so  to  speak. 

Usage  issues:  Issues  are  related  to  those  in  the  first  applicability  criterion,  as  regards  being  able  to  decode 
the  messages  of  need  or  no  need  from  industry.  A good  response  rating  could  be  interpreted  as  bad 
anticipation,  because  good  anticipation  will  obviate  the  need  for  a stated  need,  and  hence,  a response  to  that 
need.  Therefore,  if  NIST  is  perfect  in  meeting  the  anticipation  objective,  there  will  rarely  be  statements  of 
need  from  industry. 

Possible  MSID  position:  MSID  will  respond  in  timely  fashion  as  in  the  position  of  criterion  1. 

Assume: 
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• That  this  criterion  focuses  only  upon  the  nature  of  human  resources  required  and  the  status  of  the 
standard  being  considered,  while  criterion  1 focuses  on  the  origin  of  the  need  and  the  time  to  solution 

• That  the  assumptions  of  criterion  1 are  good: 

(a)  Industry  will  recognize  and  state  needs  that  exist 

(b)  needs  will  be  stated  in  a consistent  way 

(c)  Industry  will  be  as  effective  recognizing  incipient  needs  as  is  NIST 
Therefore  assume  that  no  statement  of  need  means  there  is,  and  will  be.  no  need. 

Category  3:  Does/could  NIST  use  the  standard 

Criterion  7:  Does  or  could  NIST  use  the  standard: 

Description'.  In  addition  to  the  degree  of  meeting  industrial  needs  an  added  bonus  implied  by  this  criterion 
is  whether  the  standard  helps  NIST  do  its  job  of  supporting  US  industry  better  or  more  effectively. 

Criterion  7 Breakdown: 

Since  this  is  a yes  or  no  answer,  there  is  no  breakdown 

Values  Assigned:  Does/Could  NIST  use  the  standard?  Yes  or  no.  Who  uses  it? 

Usage  issues:  Since  NIST  does  not  have  the  same  mission  or  competitive  pressures  as  industry  the  parallel 
relevance  of  this  criterion  between  NIST  and  industry  is  unclear.  Since  none  of  the  NIST  priorities  is  to 
make  NIST  more  efficient,  a high  or  low  rating  in  this  criterion  should  not  affect  the  decision  process.  This 
criterion  should  be  treated  as  a bonus.  However,  NIST  could  recommend  certain  standards  or  standard 
suites  in  its  work  with  enterprise  architectures,  frameworks,  and  manufacturing  research.  NIST  could  also 
apply  certain  standards  and  products  that  employ  standards  to  make  information-technology  aspects  of  its 
work  easier. 

Possible  MS/D  position:  NIST  will  apply  the  relevant  standards  developed  for  US  commerce. 

Assume: 

• NIST  would  only  use  a standard  if  it  is  relevant 

• NIST  use  would  not  impact  industrial  need  or  the  decision  to  help  US  industry,  and 

• NIST  use  of  the  standard  would  help  justify  industrial  use  but  NIST  non-use  would  not  impair 
justification 


4.2.2  Decision  criteria 

The  decision  criteria  mentioned  in  3.2,  above,  have  been  restated  to  be  directly  applicable  to  the  objectives 
of  the  Standards  Road-Map  project.  The  values  assigned  to  the  decision  criteria  are  intended  to  recommend 
whether  or  not  to  support  the  subject  standard.  These  decision  criteria  and  the  values  are: 

Criterion  8:  Determine  the  nature  of  the  standards  activity: 

Description:  There  are  several  ways  that  MSID  could  participate  and  it  is  important  to  define  what  is 
needed  to  be  done  so  that  the  correct  NIST  resources  can  be  identified  and  applied. 

Criterion  8 Breakdown:  No  breakdown  recommended. 

Values  assigned:  No  value  assigned,  see  assumption  below.  Determine  the  model  that  the  standard- 
development  process  is  using;  such  as: 

• National  standard;  such  as  ANSI  (American  National  Standards  Institute) 
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• International  standard;  such  as  ISO  or  IEC  (International  Organization  for  Standardization, 

International  Electrotechnical  Commission) 

• Consortium;  such  as  Consortium  for  Advanced  Manufacturing-International  (CAM-I) 

• Other  recognized  standards  body;  such  as  Institute  of  Electrical  and  Electronics  Engineers  (IEEE) 

• Program;  such  as  National  Industrial  Information  Infrastructure  Protocols  (N 1IIP) 

• Proprietary'  (eventually  to  be  open) 

This  criterion  will  be  assigned  a text  notation  of  the  type  of  development  process. 

Usage  issues : This  is  the  model  of  standard  development  rather  than  the  nature  of  the  work  covered  in 
criterion  2. 

Possible  MSID  position : NIST  will  consider  any  standards-niaking  model  to  be  relevant  that  mitigates  US 
industrial  need. 

Assume-.  Each  category  of  activity  to  develop  a standard  carries  equal  weight;  that  is,  no  rating.  This  is  an 
informative  criterion  only. 

Criterion  9:  Determine  the  need  for  NIST  resources: 

Description-.  This  criterion  is  simply  the  number  of  NIST  people  forecasted  to  be  required  to  accomplish  the 
standards  task  envisioned. 

Criterion  9 breakdown'.  No  breakdown  needed. 

Values  assigned:  Evaluate  the  need  for  resources  against  stated  needs  of  manufacturing  industry  over  the 
next  five  years.  This  criterion  will  be  assigned  a l=no  resources,  3=one  to  five  people,  or  a 5=more  than 
five  people. 

Usage  issues:  The  overall  goal  for  applying  NIST  resources  should  be  known;  that  is,  whether  it  is  more 
favorable  to  use  more  or  fewer  NIST  resources  on  a particular  standard.  This  goal  could  change  depending 
on  factors  such  as  the  type  of  standard,  the  industrial  climate  in  the  US,  and  the  availability  of  funding  in  a 
particular  fiscal  year  for  the  NIST  people.  Therefore,  this  issue  must  be  resolved  before  completing  the 
process  to  rank  a standard  using  this  criterion. 

Possible  MSID  position:  MSID  will  apply  NIST  people  to  resolve  industrial  needs  for  standards— the  more 
people,  the  better. 

Assume:  NIST  support  of  US  industry  somehow  has  been  quantified.  For  example:  applying  more  resources 
is  better  than  applying  few.  This  probably  will  be  true  except  as  changed  by  environment,  such  as  scarce 
funding. 


Criterion  10:  Determine  th z bang-for-the-buck: 

Description:  This  is  related  to  the  criterion  above  and  it  adds  a kind  of  productivity  factor  to  it  by  relating 
the  number  of  NIST  persons  to  the  desired  effect.  The  criterion  will  then  be  the  number  of  persons  per  unit 
of  desired  effect. 

Criterion  10  breakdown:  No  breakdown  needed. 

Values  assigned:  Analyze  the  bang-for-the-buck  aspect;  that  is,  how  many  NIST  people  will  it  take  to 
produce  the  desired  effect  on  the  standard  activity  and  on  the  industrial  need.  This  criterion  probably  will  be 
quantified  by  a ratio  of  NIST  investment  A divided  by  NIST  investment  B and  then  normalize  the  result 
somehow.  Sometime  during  the  process,  however,  an  analyst  must  decide  if  a ratio  greater  than  one  or  less 
than  one  is  better. 
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Usage  issues:  While  this  type  of  criterion  is  desirable,  it  is  difficult  to  partition  the  desired  effect,  largely  a 
qualitative  concept,  into  units.  A metric  should  be  identified.  Perhaps  one  way  is  to  divide  the  number  of 
NIST  people  estimated  to  produce  the  subject  standard  into  the  number  of  required  NIST  people  to 
accomplish  a set  of  deliverables  on  a similar  standard.  It  would  seem,  then,  that  a result  greater  than  1 .0  is 
more  efficient  than  a result  less  than  1 .0.  Whether  it  is,  depends  on  the  funding  climate  at  the  time.  Is  doing 
more  with  less  important,  or  is  helping  with  many  resources,  perhaps  in  differing  responsibilities  more 
important?  Another  way  is  to  separate  the  desired  effect  into  deliverables,  count  the  number  of  deliverables 
NIST  has  impact  upon,  and  compare  the  result  with  a similar  effort.  The  usage  comments  in  the  above 
criterion  about  the  possible  NIST  position  for  using  NIST  resources  also  apply  to  this  criterion.  These 
issues  must  be  resolved  before  attempting  to  rank  a standard  using  this  criterion. 

Possible  MSID  position:  Similar  to  criterion  9;  however,  MSID  will  apply  NIST  people  to  resolve  industrial 
needs  for  standards— the  more  productively  those  resources  can  be  applied,  the  better. 

Assume:  Impact  can  be  quantified  to  provide  a degree  of  impact  per  NIST  people  applied. 

4.3  Interrelationships  among  the  criteria 

The  criteria  selected  are  not  intrinsic  or  orthogonal.  Appendix  C has  been  developed  to  show  that  in  certain 
contexts  and  with  certain  degrees  of  applicability,  the  presence  of  a certain  amount  of  one  criterion  affects 
the  impact  or  the  existence  of  some  of  the  other  criteria. 

4.4  Applying  the  criteria 

4.4.1  MSID  use  of  the  criteria 

Having  the  criteria  and  the  standards  activities  the  information  must  be  navigated  in  such  a way  that  MSID 
can  extract  meaningful  decisions  from  the  information.  Making  a decision  requires  that  the  information  be 
as  complete  as  possible.  It  is  never  possible  or  worth  the  time  to  get  all  of  the  information.  Hence,  having  up 
to  date  standards  information  in  the  information  resource  will  help  a great  deal  to  create  confidence.  Even 
more  important  are  the  assumptions  and  alternatives  that  must  be  considered  when  trying  to  navigate  the 
criteria  to  assign  a numerical  score  to  a subject  standard  activity.  A sample-navigation  process  is  presented 
in  4.4.2.  This  process  is  intended  to  illustrate  the  complexity  involved  with  ranking  a candidate  standard. 

4.4.2  Combining  assumptions  and  criteria 

This  list  is  intended  to  illustrate  a process  that  could  be  used  to  evaluate  a candidate  standard  and  how  the 
criteria  must  be  navigated  using  the  relation  between  criteria  and  assumptions  using  the  composite-criteria 
breakdown.  The  items  in  the  list  are  taken  from  the  criteria  breakdowns  and  from  the  assumptions  presented 
in  4.2. 


i.  Assume  that  NIST  perceives  a need  for  a standard! s)  (criterion  1) 

ii  Assume  that  industry  can  perceive  a current  or  future  need  (criterion  1) 

iii  Assume  that  a standard  in  subject  area  corresponds  to  MSID  mission  rather  than  another  NIST 

mission  or  an  industrial  mission  that  is  not  in  NIST  purview,  (criterion  4.2) 

iv  Consider  and  score  each  of  the  four  alternatives  below: 

iv.a  Consider  if  need  is  stated  (criterion  4.2.1);  if  yes  do  criterion  1 : 

-Classify  the  need  as  large  or  small;  that  is,  the  source  of  need:  government,  industry 
segment,  industry  in  general 

-Decide,  if  need  is  not  directly  stated,  whether  NIST  has  anticipated  the  need. 
-Consider  the  immediacy  of  external  need;  that  is,  can  NIST  effectively  respond  to 
needer's  time  constraint: 

-In  less  than  three  years 
-In  less  than  five  years 

and 

iv.b  Consider  if  resources  are  available;  that  is,  is  NIST  able  to  respond  (criterion  4.2.2).  If  yes,  do 
criterion  6: 

-Assume  NIST  is  responding,  not  anticipating  as  covered  by  criterion  1 
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-Consider  standard  status  (criterion  6.2);  that  is,  degree  of  availability 
-Is  standard  needed  quickly;  say,  in  less  than  three  years: 

-Does  a Committee  Draft  exist 
-Is  this  a fast-track  opportunity 
-Is  there  a longer  time;  say,  in  less  than  five  years: 

-Has  work  started 

-Must  NIST  help  generate  consensus  regarding  need 

And 

-Consider  whether  qualified  human  resources  are  available  (criterion  6.1);  that  is,  can 
NIST  respond  effectively  to  need;  that  is: 

-Is  there  management  backing 
-Is  this  within  current  NIST  goals 
-Is  current  fiscal  climate  conducive 
-Is  sufficient  expertise  available 

and 

iv.c  Consider  the  nature  of  the  opportunity  afforded  (criterion  4.2.3);  if  acceptable  do  criterion  3: 
-Consider  the  nature  of  the  opportunity  afforded  (criterion  3) 

-Assume  that  NIST  has  found  a way  to  transfer  technology  effectively 
-Determine  what  is  required  of  NIST: 

-Technical  transfer,  and/or  (criterion  3.1) 

-Knowledge  transfer,  and/or  (criterion  3.2) 

-Publicity  (criterion  3.3) 

and 

iv.d  ■ Consider  the  opportunity  to  make  a difference  (criterion  5);  that  is,  the  nature  and  size  of 
anticipated  impact  (criterion  4.2.4).  If  positive,  do  criterion  2 

-Consider  the  nature  of  support  needed  to  include: 

-Tools  and/or 

-Standards  development,  and/or 
-Test  and  conformance  and/or 
-Capability  using  standard  and/or 
-Standards  administration 

and 

-Consider  the  size  of  the  impact;  that  is,  how  much  of  the  stated  need  is  addressed. 

v Determine  the  category  of  the  standard  (criterion  8) 

vi  Estimate  the  resources  needed  (criterion  9) 

vii  Compare  with  prior  similar  activity  (criterion  10) 

4.5  Strategic  questions  to  allow  MSID  to  use  criteria  effectively 

The  sample  navigation  of  the  criteria  in  4.4  is  an  example  of  the  process  to  evaluate  a candidate  standard 
and  to  design  a numerical  value  to  the  activity.  MSID  should  link  this  process  to  the  division  strategic-  and 
tactical-planning  activities.  Through  this  link  the  division  should  be  able  to  evaluate  its  current  goals  and 
vectors  against  the  various  different  ways  that  the  criteria  can  be  interpreted,  and  to  develop  a more 
consistent  path  through  the  criteria-option  possibilities.  These  paths  should  consider  the  issues  and 
assumptions  of  4.2  and  the  information  in  Appendixes  B and  C. 

Following  are  some  sample  strategic  questions  MSID  could  ask  to  guide  and  establish  this  consistence.  The 
items  that  follow  are  taken  from  the  discussions  in  4.2. 

1 . Impact  of  answers:  How  often  should  MSID  ask  the  following  questions  and  how  much  of  the  present 
structure  will  change  as  a result  of  the  answers? 

2.  Magnitude  of  need:  Given  that  there  is  a need,  small  or  large,  what  is  the  relative  importance  to  NIST  or 
MSID  considering  the  nature  of  the  entity  expressing  the  need;  such  as 

• Nationwide  mandate 
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• Another  government  agency 

• Industry  in  general 

• Industry  segment 

• Single  company 

3.  Immediacy  of  need:  Is  it  more  important  for  MSID  to  anticipate  a need  than  to  wait  until  a need 
becomes  immediate? 

4.  Ability  for  MSID  to  respond  in  time:  Are  sufficient  NIST  resources  available  in  the  expertise  category 
and  numbers  to  meet  the  need  satisfactorily;  that  is,  can  NIST  satisfy  the  need  with  the  FTEs  available? 

5.  Availability  of  resources:  Given  a definitive  answer  to  question  2 and  4,  what  is  the  trade-off  that  would 
enable  MSID  to  remove  resources  from  one  project  and  apply  them  to  the  next;  that  is  when  a new  project 
comes  in,  are  all  projects  reanalyzed  or  just  the  fringe  projects. 

6.  Immediacy  and  ability  to  respond:  Does  the  ability  to  respond  match  the  immediacy  of  the  need? 

7.  Nature  of  the  challenge:  What  is  the  relative  importance  of  the  different  kinds  of  standards  work  that 
needs  to  be  done,  such  as: 

• Administer  standards 

• Develop  tools 

• Develop  standard 

• Develop  test  suites,  conformance  material 

• Develop  a capability  using  the  standard 

8.  A pot  of  gold  at  the  end:  If  NIST  were  totally  flexible  and  honest  with  the  scoring  process,  will  the 
resulting  project  assignments  and  reassignments  be  a good  thing  for  standards,  MSID,  MEL,  NIST,  US? 

9.  Bang-for-the-buck:  From  the  MSID  strategic  and  tactical  plans  develop,  and  redevelop  from  time  to 
time  (item  1 above),  the  metric  to  evaluate  criteria  9 and  10.  This  will  involve  the  current  NIST 
environment  regarding  the  numbers  of  resources  that  can  or  should  be  applied  (criterion  9),  and  determining 
the  relative  productivity  and  opportunity  costs  of  applying  a certain  number  of  FTEs  (criterion  10). 

5 Computer-aided  decision  analysis 

As  a test  of  how  these  criteria  might  be  used  to  make  a decision,  the  authors  tried  a computer-aided  decision 
analysis  tool.  The  tool.  Expert  Choice  [Expert95],  is  based  on  the  analytic-hierarchy  process  developed  at 
the  Wharton  School  of  the  University  of  Pennsylvania.  The  tool  allows  pairwise  comparisons  of  the  criteria 
based  on  the  user's  understanding  and  experience.  Using  this  technique,  comparison  of  both  quantitative  and 
qualitative  concepts  can  be  done.  At  the  end  of  the  process  the  items  being  ranked,  in  this  case  each 
standard  under  consideration,  are  ordered  according  to  whatever  objective  is  selected. 

Doing  this  is  a long  process,  even  with  a few  standards  selected  for  review.  Given  the  hundreds  of  standards 
that  will  be  involved  in  an  activity  such  as  this,  a considerable  initial  time  investment  is  necessary. 
Subsequent  analyses  after  the  first  can  be  based  on  a small  portion  of  the  standards  set,  perhaps  an  analysis 
comparing  the  lowest  ranking  few  with  the  new  standards  to  be  considered. 

Individual  bias  could  be  a problem  with  some  of  these  tools.  To  continually  make  decisions  favoring  one's 
personal  preferences  will  skew  the  results.  Similarly,  if  a seemingly  unbiased  exercise  does  not  produce  the 
desired  result  the  exercise  can  be  easily  repeated  until  the  results  are  those  sought.  A well-designed  tool  will 
have  some  bias  elimination  (or  identify  that  bias  is  present)  designed  into  the  process.  If  the  preferences 
entered  were  based  on  a vote  of  interested  participants,  the  individual  bias  would  be  replaced  by  the  bias  of 
the  group-perhaps  a better  alternative. 
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Before  using  a tool  such  as  this,  the  participants  should  have  a common  understanding  of  the  meaning  of 
each  criterion  and  about  the  intended  result.  The  meaning  should  include  a scope  of  concepts  that  are 
included  and  those  that  are  not  included  in  each  criterion.  There  should  be  agreement  about  whether  a high 
score  is  favorable  or  unfavorable.  In  cases  where  the  meaning  of  a score  is  ambiguous,  the  agreement 
should  extend  to  some  assumptions  about  the  nature  of  the  score. 

These  tools  should  not  be  applied  with  the  intent  to  supplant  human  judgement  for  the  individual 
comparisons.  Managers  will  still  have  to  use  considerable  judgement  to  manage  their  resources.  The 
primary  function  of  the  tool  is  bookkeeping;  that  is,  to  do  the  math,  keep  one's  place,  apply  the  attributes  of 
the  decision  in  an  orderly  way,  and  to  record  preferences.  The  result  will  have  used  judgement  and 
bookkeeping  to  allow  managers  to  combine  analysis,  relevant  information,  knowledge,  and  experience  into 
a higher  confidence-level  result  than  intuition  alone  could  produce. 
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6 Conclusions 


The  criteria  are  adapted  to  manufacturing  situations  from  the  priorities  set  forth  in  Setting  Priorities  and 
Measuring  Results  at  the  National  Institute  of  Standards  and  Technology > [BELL094].  There  are  two  types 
of  criteria. 

• Applicability  criteria  help  assess  the  standard  in  question  with  respect  to  MSID  work.  Applicability 
criteria  arrange  into  three  categories:  applicability  to  the  NIST  mission,  the  relevance  for  MSID 
involvement,  and  the  usefulness  of  the  standard  to  NIST. 

• Decision  criteria  lead  directly  to  a decision  about  whether  MSID  should  apply  resources,  how  much 
resources,  and  about  the  ability  of  NIST  to  apply  resources  effectively. 

Quantifying  the  criteria  requires  careful  planning  regarding  what  a particular  ensuing  score  might  mean. 

The  criteria,  once  restated  in  a manufacturing  context,  must  be  analyzed  to  assure  understanding  about  what 
a high  score  means,  what  a low  score  means,  and  what  is  the  significance  of  different  scores. 

Most  of  the  criteria  can  be  broken  into  subcriteria.  Before  applying  the  criteria,  certain  assumptions  must  be 
made  about  the  environment.  These  assumptions  are  then  used  with  each  sub  criteria.  This  process  should 
be  organized  to  form  a road  map  for  navigating  the  criteria  that  will  ultimately  lead  to  a strategic  decision 
about  assigning  a priority  to  a standard. 

The  goal  of  applying  the  criteria  is  to  be  able  to  bring  more  structure  to  this  aspect  of  the  division  strategic- 
planning  process  and  more  knowledge  about  the  relative  value  of  particular  standards  activities.  Therefore, 
for  the  strategic-decision  process  relative  to  standards  to  be  more  effective,  MSID  should  insert  some 
strategic  questions  into  the  strategic-planning  process  to  provide  a current  context  for  the  decisions. 

A computer-aided  decision  tool  will  help  analysts  to  manipulate  the  amount  of  variables  that  arise  when 
evaluating  the  standards  using  the  criteria  and  assumptions. 


17 


7 References 


[BELL094]  Mark  Bello  and  Michael  Baum,  NIST,  Setting  Priorities  and  Measuring  Results  at  the 
National  Institute  of  Standards  and  Technology’,  1994-January,  page  19. 

[CEN96]  Committee  for  European  Standardization,  TC310  Strategic  Working  Group,  Standardization  for 
Advanced  Manufacturing  Technologies,  Memorandum  M-IT-04,  issue  6,  parts  1 and  2. 

[Expert95]  Expert  Choice,  Decision-Support  Systems,  Tutorial  Document,  Version  9,  Decision  Support 
Systems,  Inc. 

[ISO90]  International  Organization  for  Standardization,  Reference  Model  for  standardization  and 
methodology  for  identification  of  requirements,  TR  10314  1,  1 990-October. 

[1S091]  International  Organization  for  Standardization,  Reference  Model  for  standardization  and 
methodology’ for  identification  of  requirements,  TR  1 03 1 4 2,  1991  -June. 

[NELL97]  James  G.  Nell,  NIST,  A Standardization  Strategy  that  Matches  Enterprise  Operation , NISTIR 
6049,  1997-September. 

[NELL99]  James  G.  Nell  and  Neil  B.  Christopher,  NIST,  Standards  Classification  Strategy > and 
Methodology  Standards  Road  Map  Project,  To  be  published 

[THOM97]  Kristy  D.  Thompson  and  Cynthia  K.  Montgomery,  NIST,  Program  of  the  Manufacturing 
Engineering  Laboratory’,  NISTR  5968,  1997-February,  page  182. 

(ROBY96]  Clyde  G.  Roby,  Institute  for  Defense  Analyses,  Evaluation ; Models  for  Standards  and 
Products,  IDA  document  D-1840,  1996-May. 


18 


Appendix  A:  Listing  of  criteria  for  the  standards  road-map  project 

Sample  values  added  for  illustrative  purposes 


Criterion  name 

Values 

Explanation  of  criterion 

Applicability  criteria 

1 . Industrial  need 

Within  3 yr.  5 
Within  5 yr.  3 
Not  Stated  1 

Magnitude  and  immediacy  of 
industrial  need 

2.  MSID  impact 

Direct  5 

Indirect  3 

Slight  1 

Nature  and  size  of  anticipated  impact 
resulting  horn  NIST  participation 

3.  Technical  transfer 

Topical  5 

Average  3 

Not  relevant  1 

Nature  of  opportunities  afforded  by 
recent  advances  in  science  and 
technology 

4.  Relevance 

Direct  5 

Indirect  3 

Slight  1 

Degree  of  correspondence  between  a 
particular  industrial  need  and  NIST 
mission  to  develop  infrastructure 
technologies 

5.  SOTA 

Direct  5 

Indirect  3 

Slight  1 

Opportunity  for  MSID  to  affect  the 
state-of-the-art  of  the  technology  or 
the  standard 

6.  Response 

Anticipatory  5 
With  Tech.  3 
Reactive  1 

NIST  capability  to  respond  in  timely 
fashion  with  high-quality  solutions 

7.  NIST  use 

Yes 

No 

Is  it  relevant  for  NIST  to  use  the 
standard,  and  does  NIST  use  it 

Decision  criteria 

8.  Approach 

Standard 

Test  Cases 

Test  Certify 

Tools 

Nature  of  the  effect  desired  or 
required 

9.  NIST  FTEs 

Yes 

No 

Should  NIST  assign  resources  and 
how  many 

10.  Bang-for-buck 

>5  5 

1-5  3 

None  1 

Number  of  FTEs  required  to  achieve 
desired  effect 
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Appendix  13:  Interpretations  possible  when  evaluating  the  criteria  with  respect  to 

the  standards 

Criterion  1:  Magnitude  and  immediacy  of  industrial  need 


Issue 

Possible  interpretation 

Solution  or  comment 

Industry'  has  no  direct  need  for 
standards 

• No  consistent  answer  re 
standards  needs 

• NIST  must  translate 

standards  development  into 
industry  people's  real  need 

Industry  has  no  opinion,  specific 
people  do 

• People  want  solution  to 
needs  that  reward  them-- 
lower  cost,  higher  quality 

• NIST  must  translate 
standards  development  into 
an  individual's  real  need-to 
make  parts 

• Also  address  the  need  for  a 
balanced  development 

What  if  there  is  a "stated  need" 

• There  is  a need  and  NIST 
can  help 

• It  may  be  too  late  to  help 

• NIST  must  assume  or 
analyze  situations  on  a case- 
be-case  basis  and  help  where 
there  can  be  impact 

• Too  late  means  that  NIST 
cannot  have  timely  response 
per  criterion  6 

What  if  there  is  "no  stated  need" 

• There  is  a need  and  solution 
is  being  sought 

• There  is  no  need 

• There  is  no  need  yet 

• The  need  has  been  met 

• NIST  must  assume  or 

analyze  situations  on  a case- 
be-case  basis  and  help  where 
there  can  be  impact 

Comment 

• Needs  should  be  those  that 
are  pre-competitive  or 
infrastructural  that  does  not 
preclude  solution  by  private 
sector 

Criterion  2:  Nature  and  size  of  anticipated  impact 


Issue 

Possible  Interpretation 

Solution  or  comment 

Can  interpret  a value  that 
indicates  NIST  could  have: 

• Slight  impact 

• A small  effect  on  small 
problem 

• A small  part  of  key  large 
problem 

• Identify  the  domain  and  the 
value-added  to  the  US 
industry  as  a whole  that 
warrants  NIST  support 

• Major  impact 

• A large  part  of  insignificant 
problem 

• Large  part  of  key  large 
problem 

• Identify  the  domain  and  the 
value-added  to  the  US 
industry  as  a whole  that 
warrants  NIST  support 

To  what  degree  is  US  government 
involved  in  US  and  international- 
standards  making 

• Standard  could  become  too 
government 

• Problem  is  not  general,  hence 
it  could  benefit  a few 
companies  at  a cost  to  others 

• Identify  the  domain  and  the 
value-added  to  the  US 
industry  as  a whole  that 
warrants  NIST  support 

Appendix  B continued 
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Criterion  3:  Nature  of  opportunities  afforded  for  technical  transfer 


Issue 

No  issues  except  to  design  a technical-transfer  methodology  that  works 


Criterion  4:  Degree  of  correspondence  between  industrial  need  and  MSID  mission 


Issue 

No  issue  except  that  we  assume  that  industrial  need  is  not  in  conflict  with  MSID  mission 


Criterion  5:  Opportunity  for  MSID  to  make  a difference. 


Issue 

No  issue;  however  recommend  that  this  criterion  be  combined  with  criterion  2:  Nature  and  size  of 
anticipated  impact 


Criterion  6:  Ability  to  anticipate  and  respond  in  a timely  fashion. 


Issue 

Possible  Interpretation 

Solution  or  comment 

Assuming  that  there  is  a real  need 
per  criterion  1,  can  NIST  help? 

• NIST  can  respond  in  timely 
way 

• NIST  has  the  expertise 

• Standard  is  in  late  stages  of 
its  development 

• Standard  in  late  stages  of 
development  may  not  permit 
NIST  an  acceptable  size  and 
nature  of  impact,  per 
criterion  2 

• NIST  cannot  respond  in 
timely  way 

• Standard  is  in  early  stages  of 
its  development 

• NIST  lacks  the  expertise 

• Reevaluate  the  meaning  of 
the  criterion 

Anticipation  and  response  may  be 
mutually  exclusive 
• Good  anticipation 

• Obviate  stated  need  and  need 
for  response 

• Bad  anticipation 

• Opens  opportunity  for  good 
response 

• Good  response 

• NIST  is  flexible,  but  not 
anticipatory 

• Bad  response 

• NIST  is  neither  anticipatory 
nor  responsive 

General  Issue:  This  addresses 
MSID  human  resource  and 
organizational  aspects  that  are 
related  to  the  issues  of  criterion  1 : 
Magnitude  and  immediacy  of 
industrial  need 

• Not  being  able  to  respond 
impacts  the  quality  of 
response  to  criterion  1, 
immediacy  of  need 
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Appendix  B continued 

Criterion  7:  Does  or  could  NIST  use  the  standard? 


Issue 

Possible  Interpretation 

Solution  or  comment 

• NIST  uses  the  standard 

• It  is  a good  standard 

• Good  or  not  it  suits  NIST 
purpose 

• It  is  the  only  standard  of  its 
kind  available 

• NIST  use  is  not  a 

determinant  but  non-use 
where  the  standard  would 
apply  could  have  a negative 
impact  on  general  use. 

• NIST  does  not  or  cannot  use 
the  standard 

• Not  relevant  to  NIST  work 

• Not  a good  standard. 

• NIST  prefers  a competitor 

• NIST  use  is  not  a 

determinant  but  non-use 
where  the  standard  would 
apply  could  have  a negative 
impact  on  general  use. 

Is  it  valid  to  assume  that  NIST 
would  use  a standard  only  if  it 
added  value  to  NIST  mission? 

• NIST  use  or  non  use  does  not 
translate  directly  into  an 
industrial  model. 

• NIST  use  makes  no  statement 
about  efficient  applicability 
to  industrial  use.  NIST  has 
no  explicit  goal  to  improve 
its  efficiency  in  this  area. 

Criterion  8:  Determine  the  nature  of  the  standards  activity. 


Issue 

Possible  Interpretation 

Solution  or  comment 

• This  is  an  information-only 
criterion  because  the  relevant 
standard  organizations  are 
perceived  to  be  equal. 
Therefore  it  is  not  a criterion 
but  a data  item. 
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Appendix  B continued 

Criterion  9:  Determine  the  need  for  NIST  resources. 


Issue 

Possible  Interpretation 

Solution  or  comment 

• Uses  few  resources 

• Efficient  use  of  resources 

• Insufficient  support-will  not 
meet  need  in  time 

• Sufficient  or  insufficient 
govemment/industry  balance 

• These  two  issues  must  be 
evaluated  with  respect  to 

NIST,  MEL,  and  MS1D 
funding  and  objectives  that 
are  in  place  at  the  time  of 
analysis 

• Uses  many  resources 

• Good  and  sufficient  support 

• Sufficient  or  insufficient 
govemment/industry  balance 

• It  may  be  difficult  to  evaluate 
a standard  based  on  whether 
or  not  many  or  few  resources 
are  used 

Availability  of  NIST  standards 
experts  will  be  a determining 
factor  in  all  standardization 

• With  or  without  the  various 
interpretations,  this 
consideration  may  be  the 
ultimate  and  sole  determinant 
of  support  for  a particular 
standard 

Criterion  10:  Determine  the  bang-for-the-NIST  buck. 


Issue 

Possible  Interpretation 

Solution  or  comment 

This  is  an  attempt  to  quantify 
criterion  9:  Determine  the  need 
for  NIST  resources 
• Difficult  to  partition  this  into 
measurable  units 

• Comparing  one  standard  with 
another  may  give  an 
impression  that  the  process  is 
totally  meaningful  (apples 
and  oranges) 

• Could  devise  a metric  such 
as:  Divide  the  FTEs  required 
to  produce  deliverables  on 
the  subject  standard  by  the 
FTEs  required  to  accomplish 
similar  deliverables  on  a 
similar  standard. 

• Is  high-ranked  metric  better 
or  worse  than  low-ranked 
metric? 

• NIST  can  be  evaluated 

highly  or  poorly  by  applying 
many  or  few  resources  per 
unit  of  output 

• Whether  or  not  the  indicators 
are  good  will  depend  on  the 
climate  as  discussed  on 
criterion  9 
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Appendix  C:  Interrelationships  among  applicability  criteria  (See  4.5) 


I Industrial 
need 

2 MSID 
impact 

3 Technical 
transfer 

4 Relevance 

5 State-of- 
the-art 
(SOTA) 

6 Anticipate 
or  respond 

1 Industrial 

need 

X 

Requires 

accurate 

account  of 
need.  Need 
may  become 
more 

immediate  if 
MSID  can't 
respond. 

Communi- 

cate 

anticipated 
needs  to 
industry  may 
create  a new 

need  and 
justify  more 
support. 

None  direct, 
if  assume 
industrial 
needs  as 
stated  to 

NIST  are 
infrastruc- 
tional  in 

nature. 

Improved 
SOTA  will 
spawn  new 
categories  of 
need. 

Requires 

accurate 

account  of 
need.  Need 
gets  more 
immediate  if 
MSID  can't 
respond. 
Anticipation 
receives  no 
statement  of 
need. 

2 MSID 
impact 

X 

X 

Impact  is 
achieved 
only  through 
technical 
transfer.  The 
weaker  the 
transfer,  the 
weaker  the 
impact. 

These 
should 
correlate  for 
highest 
impact. 

Impact  will 
attract  R&D 
to  improve 
SOTA. 

These  two 

should 

correlate. 

Being  able 
to  anticipate 
may  lessen 
perceived 
impact.  A 
good 
response 
will  improve 
perceived 
impact. 

3 Technical 
transfer 

X 

X 

X 

Technical 
transfer  is 
better  when 
relevant  to 
both  parties. 

Good 
technical 
transfer  will 
impact 

SOTA. 

Quality  work 
improves 
receptivity 
for  tech, 
transfer. 

4 Relevance 

X 

X 

X 

X 

These  two 

should 

correlate. 

Process 
improves 
with  proper 
MSID 

resources 

5 State-of- 
the-art 

X 

X 

X 

X 

X 

Good 

anticipation 
or  response 
should 
improve 
SOTA. 

6 Response 

X 

X 

X 

X 

X 

X 
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Appendix  D:  Criteria  listing  divided  into  their  meaning  segments 

Applicability  criteria 
Category  1 

1 Magnitude  and  immediacy  of  industrial  need 

1 . 1 Magnitude  of  industrial  need 

1 . 1 . 1 Nature  of  entity  expressing  need 

1 . 1 . 1 . 1 Congress-new  law 

1 . 1 . 1 .2  Industrial— segment 

1 . 1 . 1 .3  Industry— general 

1.2  Immediacy  of  industrial  need 

1.2.1  NIST  anticipated  the  need 

1 .2.2  Ability  to  respond 

1.2. 2.1  Industry  needs  benefit  in  less  than  three  years 

1. 2.2.2  Industry  needs  benefit  in  less  than  five  years 

2 Nature  and  size  of  anticipated  impact. 

2. 1 Nature  of  impact 

2.1.1  Nature  of  support  needed 

2. 1.1.1  Develop  tools 

2. 1 . 1 .2  Develop  standard 

2. 1 . 1 .3  Develop  test,  conformance 

2. 1 . 1 .4  Develop  capability  that  uses  standard 

2. 1.1. 5 Standards  administration 

2.2  Size  of  impact 

2.2.1  Impact  in  small  part  (less  than  20%)  of  problem  (stated  need  ) 

2.2.2  Impact  in  large  part  (more  than  20%)  of  stated  need 

3 Nature  of  opportunity  afforded 

3.1  Technology  transfer 

3.1.1  Seminars 
3.1 ,2Documents 

3.1.3  Co-participation 

3.2  Knowledge  transfer 

3.2.1  Books 

3.2.2  Knowledgebase 

3.3  Publicity 

3.3.1  Mass 

3.3.2  Focused 

Category  2 

4 Degree  of  correspondence  between  industrial  need  and  MSID  mission 

4.1  Industry  mission 

4.2  MSID  mission 

4.2.1  Stated  industry  need  (criterion  1) 

4.2.2  Availability  of  resources  (criterion  6) 

4.2.3  Nature  of  opportunity  afforded  (criterion  3) 

4.2.4  Nature  and  size  of  anticipated  impact  (criterion  2) 

4.3  Other  mission 

5 Opportunity  for  MSID  to  make  a difference  (covered  in  4.2  and  becomes  criterion  2) 

6 Ability  to  anticipate  and  respond  in  timely  fashion 
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6.1  Human  resources  available 

6.1.1  Degree  of  management  support  or  commitment 

6. 1 .2  Current  NIST  goals 

6. 1 .3  Current  fiscal  climate 

6. 1 .4  Expertise  level  of  resource 

6.2  Standard  status-Availability  of  standard 

6.2. 1 Short  time  until  use 

6. 2. 1.1  CD  exists 

6.2.1 .2  Draft  standard  prepared  and  balloted 

6.2.2  Long  time  until  use 

6.2.2. 1  Work  not  yet  started--N  1 ST  to  help  generate  consensus  re  need  for  standard 

6.2.2. 1 . 1  NIST  to  generate  consensus  regarding  need  for  standard 

Category  3 

7 Does  or  could  NIST  use  the  standard  (No  breakdown  recommended) 

Decision  criteria 

8 The  nature  of  the  standards  activity  (No  breakdown  recommended) 

9 The  need  for  NIST  resources  (No  breakdown  recommended) 

10  The  bang-for-the-buck  (No  breakdown  recommended) 
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Appendix  E:  Graphic  organization  of  the  criteria 


Criterion  1:  Magnitude  and  immediacy  of  industrial  need 


1.1.1  Nature  of 
entity  expressing 
need 


1.2  1 NIST  1.2  2 Ability 

anticipation  to  respond 

(no  stated  need)  j j (stated  need) 


1.1.1  1 Congress 
new  law 


1.1. 1.3  Industry 
general 


1.2. 2.2  Industry 
<5  years 


Criterion  2:  Nature  and  Size  of  Anticipated  Impact 
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Appendix  E continued 


Criterion  3:  Nature  of  Opportunity  Afforded 


Criterion  4:  Degree  of  correspondence 
between  need  and  MSID  mission 
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Appendix  E continued 


Composite:  criteria  1,  2,  3,  5 , anti  6 positioned  under  criterion  4 


Criterion  6:  Ability  to  anticipate  or  respond  in  timely  fashion 


42  2 Ability 

to  perform 


6.  Ability  to  perform  j 
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Appendix  E continued 


Criteria  7,  8,  9,  and  10:  Decision-criteria  composite 
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